The present invention relates to digital imaging technology, and more particularly to contrast and edge enhancement in still and moving digital imaging.
Digital imaging offers a great many advantages over traditional analog techniques, supporting services such as video telephony and multi-media applications. Accordingly, digital imaging is currently the subject of extensive research.
One particular area of digital imaging research is that of image enhancement. In digital imaging, there is need for image enhancement because images typically suffer from various types of degradation, such as lack of sharpness, noise, compression artefacts, and poor contrast and brightness. Mobile imaging products presently under development are especially prone to such degradations due to the low performance of current digital camera devices and the narrow data transmission bandwidth typically available in mobile communications networks, which requires the use of high compression ratios. In digital camera products, poor image quality arises from shortcomings in fundamental sensor properties. The lack of contrast typically arises from the sensor itself, while the lack of sharpness is the result of both low quality optics and low sensor resolution. Low resolution sensors are likely to be used in the imaging systems of forthcoming first generation mobile telecommunication terminals with image presentation and manipulation functionality. Thus, there is a need for methods to enhance image contrast and sharpness, and in current digital image enhancement techniques this is commonly carried out by concealing artefacts and reconstructing degraded parts of an image.
Image enhancement principally concerns accentuation of image features such as edges, boundaries, or contrast. This makes image details more visible and makes images look subjectively sharper or crisper without appearing unnatural, therefore making a graphic display more useful for display and analysis. Whilst the enhancement process does not increase the inherent information content in the data, it does increase the dynamic range of chosen features so that they can be more easily detected. Image enhancement techniques can be divided roughly into three sub-classes comprising point, spatial, and transform operations.
Point operations are performed on individual pixels and as such are independent of surrounding pixel values. Pixel mapping operations are one example of point operations. Pixel mapping involves mapping a pixel value to a new value with a function, for example:yi,j=f(xi,j),where    i,j=pixel's coordinates    y=output pixel    x=input pixel    and f is a function relating x and y.
Spatial operations are performed on groups of pixels. Certain image smoothing and filtering functions are spatial operations where, for example, a pixel's value may be modified by taking an average of surrounding pixel values.
Transform operations convert image data from pixel values into another form, for example spatial frequency components. Discrete cosine transforms (DCT) and wavelet transforms are examples of transform operations commonly used in image processing applications.
Examples of enhancement techniques in the sub-classes outlined above include techniques such as grey level and contrast manipulation, noise reduction, edge crispening and sharpening, filtering, interpolation and magnification, and pseudo-coloring.
In practice, images taken by a digital camera normally suffer from smoothness problems irrespective of the pixel resolution. This means that there is always likely to be a need for image sharpening techniques. Image sharpening can be performed by applying various forms of contrast enhancement (stretching/shrinking) and edge enhancement techniques. Contrast enhancement is a point operation that can be used to enhance images taken in poor or nonuniform lighting conditions, or acquired with an imaging sensor having a small dynamic range. Contrast manipulation also tends to alter the visibility of edges, which changes the perceived sharpness of the image. Edge enhancement techniques are typically spatial operations and they enhance images that suffer from overall smoothness. A technique known as unsharp masking is an example of an edge enhancement technique.
Referring first to contrast enhancement, a well known method for enhancing contrast is histogram-based pixel mapping. In this method a pixel-mapping function is formed utilizing a histogram of the image, or a part of an image. The histogram contains the number of pixel occurrences at each level of the input signal range. The mapping function is derived from a cumulative histogram so that it has high slope corresponding to high values in the histogram. The cumulative histogram is typically formed using an equation having the form:
            C      i        =                  ∑                  k          =          0                i            ⁢                          ⁢              H        k              ,where    Ci=bin i of cumulative histogram    Hk=bin k of histogram
Note that a bin is a pixel value range, that may cover a single pixel value or more than one pixel value.
In the basic method the mapping function is:f(x)Cx
The mapping function or the cumulative histogram is typically scaled so that the mapping covers substantially exactly the available dynamic range, i.e.:f(L)=L,where L=largest value of the available dynamic range
The scaling is achieved, for example, using equation:
      f    ⁡          (      x      )        =      L    ⁢                            C          x                          C          L                    .      
The above equations can be found in slightly varying forms in many signal processing textbooks. For example, in one alternative approach the first nonzero bin is subtracted from the histogram before or during computation of the cumulative histogram. If such a subtraction is not performed the darkest pixel value of the image is not mapped to zero (C0?0). Other modifications also exist.
The above equations are usable substantially only if there are as many bins as signal levels, for example, 256 bins are required for a signal with 8 bits/sample. If fewer bins are used, each bin contains pixels belonging to a certain signal range. The above equations provide mapping function values at the borders of bins. Additional values may be obtained by interpolation.
Examples of a histogram, a cumulative histogram and corresponding mapping function are presented in FIG. 1.
Using histogram-based pixel mapping techniques, the contrast of an image can be increased by mapping input pixel values to output pixel values with a linear function having a slope (i.e. derivative) greater than one. A slope less than one decreases contrast, while a slope equal to one produces no change in contrast. The mapping function can also be non-linear, having a varying slope. In this case contrast around some pixel values of the image is increased, the contrast in other pixel value ranges may be decreased or not altered at all. An example of a non-linear mapping function is shown in FIG. 2.
An acknowledged limitation of histogram-based pixel mapping is that it tends to make noise and other artefacts disturbingly visible. Because increasing contrast increases the visibility of variations in the input signal, if there is noise present in the input signal, its visibility is also increased. Because of this, histogram-based pixel mapping is often considered unsatisfactory for contrast enhancement of natural images.
Nevertheless, histogram-based pixel mapping is the basis for a substantial number of contrast enhancement algorithms. So, whilst these algorithms form a mapping function based on the image histogram, attempts are made to mitigate the unwanted effects of the basic mapping procedure using modifications and additions to the basic algorithm. These modifications include, for example, reshaping the histogram, reshaping the mapping function, filtering the image, the histogram or the mapping function and limiting the maximum value of the histogram.
Turning now to edge enhancement, specifically in the context of unsharp masking, this is a spatial filtering technique for improving the visibility of edges and thus improving perceived image quality. Unsharp masking is commonly implemented in the form of a highpass filter in series with an adjustable gain block, as shown in FIG. 3. In the illustrated arrangement, a highpass filtered and scaled image signal is added to the original image signal as shown in FIG. 4a. Thus, the desired improvement is achieved by emphasizing high frequency components of the image. Specifically, the output of the highpass filter represents a second order derivative of the original signal, as represented in FIG. 4b. When the highpass filtered signal is added back to the original signal, it creates under and overshoots around edges as shown in FIG. 4c. The human visual system itself creates similar under and overshoots around sharp edges. This phenomenon is known as the Mach band effect. It demonstrates that the human visual system perceives edges in a non-linear manner, and that perception depends not only on luminance but also on image structure. For this reason, the human visual system perceives edges with added under and over shoot, such as those produced by unsharp masking, as sharper edges.
Unsharp masking also tends to produce some undesirable effects. The presence of a highpass filter makes the system sensitive to noise and compression artefacts. Thus, unsharp masking typically produces perceivable artefacts, which are especially visible in uniform areas of an image.
Accordingly, it has been recognized that in order to improve image quality it would be appropriate to modify an input signal of the highpass filter, especially when there is a noise component present in an image. One such modification, known in the art, is to use a standard, predefined, or fixed pixel mapping function prior to unsharp masking. Using a conventional fixed pixel mapping function, it is possible to emphasis certain edges, whilst limiting unwanted noise enhancement. FIG. 5 depicts an unsharp masking scheme that utilizes pixel mapping prior to highpass filtering. In the illustrated arrangement, a pixel mapping function is applied to the image before unsharp masking in order to limit the unwanted enhancement of noise components that occurs as a result of high-pass filtering.
FIG. 6 shows two possible pixel mapping function curves suitable for use prior to unsharp masking. Curve 1 suppresses high and low level signals and stretches mid-range pixel values. Accordingly, in the highpass filter, high and low level edges are suppressed and mid level edges are enhanced. In Curve 2 mid level edges are suppressed and high and low level edges are enhanced. In this way, the effect of using pixel mapping prior to highpass filtering is to reduce noise.
The effect of a function such as Curve 1, shown in FIG. 6, is illustrated in FIG. 7. This shows the effect of applying a conventional fixed pixel mapping function to an image that contains a single ‘edge’. Here the edge is a transition from dark to light i.e., from very low intensity pixel values to very high intensity pixel values. In other words, the image contains just two regions, one of low luminance values and one of high luminance values, as shown in FIG. 7a. FIG. 7b presents a graph of pixel values created by scanning across a row of pixels from left to right. It is assumed that the two regions, which should be of essentially uniform intensity, exhibit some degree of noise. The noise manifests itself as small variations in pixel value (as can is be seen from the ‘ripple’ in FIG. 7b). By applying the fixed pixel mapping function illustrated in FIG. 7c, the noise in the image of FIG. 7b can be suppressed effectively. The result of the fixed pixel mapping function is illustrated in FIG. 7d. The way in which the noise is suppressed can be understood by examining the pixel mapping function in detail. It should be appreciated that the pixel mapping function describes the way in which pixel values in the original image (i.e., that described by FIG. 7b) are converted into pixel values in the contrast enhanced (adjusted) image, whose pixel value profile is illustrated in FIG. 7d. Considering first reduction of noise in the low intensity region of the image, the relevant part of the pixel mapping function is that labeled ‘A’ in FIG. 7c. This part of the pixel mapping function determines how low intensity pixel values in the original image are translated into low intensity pixel values in the contrast enhanced image. Here the pixel mapping function has a gradient less than 1, indicating that variations in pixel value will be reduced (or ‘compressed’) by applying the function. The same is true in region ‘B’ of the pixel mapping function of FIG. 7c. This part of the curve determines the way in which high intensity pixel values in the original image are translated into high intensity pixel values in the contrast-enhanced image. Thus, noise is effectively reduced at both extremes of the range. Referring to how mid-range intensity values are converted by the pixel mapping function in this example, the part of the function labeled ‘C’ has a gradient greater than 1 and therefore, the range of pixel value variation in this region is actually enhanced (expanded or stretched), changing the profile of the transition between low and high intensity values. However, this effect cannot be considered as a true sharpening of the transition, such as that provided by unsharp masking. Contrast expansion has the effect of enhancing the difference between nearly identical mid-range pixel values and inevitably enhances the effect of noise in the pixel value range around the transition.
The use of conventional fixed pixel mapping functions in order to condition an image prior to unsharp masking has certain limitations. These will be described with reference to FIG. 8, which illustrates the effect of using the same pixel mapping function, introduced above, to enhance the contrast of a second example image. As in the first example, the image considered in FIG. 8 is a very simple one, comprising two substantially uniform regions, one darker than the other (FIG. 8a). The difference in intensity values between the two regions is less than in the previous example, although the same kind of noise is present (FIG. 8b). The effect of the fixed pixel mapping function is illustrated in FIG. 8d. While noise in the darker region is suppressed, noise in the lighter region is actually enhanced. This effect occurs because the pixel values of the lighter region correspond to a region of the pixel mapping function that has a gradient greater than one. Thus, small differences in pixel value are enhanced, rather than suppressed, causing an increase in noise.
It is common that images are processed using both (fixed or histogram-based) pixel mapping contrast algorithms and unsharp masking edge enhancement algorithms. Referring to FIG. 9, the combination of histogram-based pixel mapping and unsharp masking can be implemented by placing the histogram-based pixel mapping in the original image signal branch before (Option A), after (Option C), or in parallel (Option B) with an edge enhancement unit. Commonly, image processing applications, such as certain commercially available software packages, implement options A or C, giving rise to a cascaded structure. However, it has been found that the best results are achieved by option B, in which a histogram-based pixel mapping unit is placed inside the unsharp masking unit, that is in the original image signal branch, in parallel with the highpass filter and not in cascade with it as in options A and C. In option B, any unwanted noise component that is emphasized by contrast enhancement, is not highpass filtered and thus not further emphasized. In both options A and C the second operation in the cascaded structure further emphasizes noise artefacts introduced by the first operation.
Here, the second operation refers to the latter operation, i.e., in option A, the second operation is unsharp masking, and in option C the second operation is histogram-based pixel mapping. Thus, the cascaded approach has the disadvantage that noise is ‘doubly’ enhanced. This can create very annoying noise structures, especially if a JPEG or DCT based hybrid codec has been used to compress the original source image. For example, assuming option A is used, initial contrast enhancement causes a certain increase in pixel value noise. This is further exaggerated by the following step of edge enhancement using e.g., unsharp masking. In the case of option C, because the unsharp masking emphasizes edges it also heightens noise around edges. The amplitude of this noise is then further increased by the histogram-based contrast enhancement. Also, other artefacts caused by the unsharp masking, for example ringing effects may become more visible. The approach illustrated by option B, on the other hand, avoids double noise enhancement. Here, contrast and edge enhancement are performed in parallel and the high pass filtered image is added to the contrast enhanced image (as indicated by the addition symbol in FIG. 9).
Nevertheless, the combined image enhancement options of FIG. 9 suffer from the problems associated with contrast and edge enhancement previously described above.